DOCUMENT RESUME 



ED 324 213 * SE 051 592 

TITLE The North Carolina Test of Chemistry. Technical 

Characteristics* Forms A-D. 
INSTITUTION North Carolina State Dept. of Pul>lic Instruction, 

Raleigh. Div. of Accountability Services/Research. 
PUB DATE 90 
NOTE 42p- 

PUB TYPE Reports - Descriptive (141) — Statistical Data (110) 

EDRS PRICE MF01/PC02 Plus Postage. 

DESCRI^'^RS *Chemistry; *Curriculum Evaluation; Science 

Education; Secondary Education; *Secondary School 
Science; *Test Construction; Test Content; *Testing 
Programs; Test Norms; Test Reliability; *Test Theory; 
Test Validity 

IDENTIFIERS *North Carolina Test of Chemistry 



ABSTRACT 

To facilitate the proper technical use of the test 
scores obtained from the administration of the tests, the curricular 
and psychometric characteristics of the tests are described in a 
series of technical manuals. This laanual, the seventh in the series, 
contains a description of the characteristics of the North Carolina 
Test of Chemistry. The test was developed for use as an achievement 
test following the completion of the Chemistry course of study. Its 
design was to serve as a normative measurement of student achievement 
and as an objective-based measurement of curriculum coverage. The 
curricular, irstructional, content, and concurrent validity of tlie 
test are dxscussed. Specific issues covered includes (1) the method 
for deriving test scores; (2) reliability and other statistics; (3) a 
curriculum assessment; (4) content of the test; and (5) test norms. 
The utility of the test has 'sj^en determined by its statistical 
equivalence of core tests from year to year, its broad sampling of 
the curriculum across time, and its initial norms table. The goals 
and objectives used to assemble the test and the number of questions 
associated with each is appended. Th"^ percent of teachers reporting 
each goal/objective as basic to instruction in Chemistry as well as a 
list of goals/objectives that were rejected for use are also 
appended . (KR) 



* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 
*********************************************************************** 



Forms A- D 

CO 
Q 



Technical Characteristics of the 

North Carolina Test of 

Chemistry 



fl) 

k> 



M/VTERIAL HAS BEEN GRANTED BY 




RESOURCES 



TO THE EDUCATIONAL BESUU 
INFORMATION CENTER (ERIC). 



SJwpotilonorpoficy. 



NorthCarolinaDepartoentofPublictetoKHon 
Division of AccountabiHty Setvices/Reseaich 
Raleigh, NC 27603-'i332 



NCTests 




V 




Technical Characteristics of the 

North Carolina Test of 

Chemistry 



Prepared by 
Eleanor E. Sanford 



NCTosts 



North Carolina Department of PubHc Instruction 
Division of Accountability Services/Research 
Raleigh, NC 27603-1332 



FOREWORD 



The NCDPI Division of Accountability Services/Research, in coopferaHon with the NCDPI 
Division of Curriculum and Instruction Services, has devdoped diagnostic achievement tests 
of basic skills for public school students in Grades 3, 6, and 8; survey achievement tests of 
Science and Sodal Studies for students in Grades 3, 6, and 8; and high-school course 
achievement tests for students taking Algebra I, Algebra n. Biology, Chemistry, English I, 
Geometry, Physics, and U.S. History. Physical Science and Economic, Legal, and Political 
Systems will be added in 1991, and other tests are being planned." 

To facilitate the proper technical use of the test scores obtained from the administration of the 
tests, the curricular and psychometric characteristics of the tests are described in a series of 
technical manuals. This manual, the seventh in the series, contains a description of the 
characteristics of the North Carolina Test of Chemistry. 



"Readers who have an interest in the origins of the test development program are referred to the North Carolina 
Elementary and Secondary Reform Act of 1984, the North Carolina Basic Education Program, the North Carolina 
Standard Course of Study, and the Teacher Handbook. 
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Description 

Tli^NoithCarolinaTestofChem^ 

test foUowing the cojc^pletion.of the Chenustiy <x»^^^ 

purpose: that of a normative measur«nent of student achievement and of ari obtefctive-ba^ 
measiurement of ciirriculum covei'age. 

l^e measurement of student acWevement is attained by a^ 

to all students. The measurement of curriculum is met by an additional 40 items that vary 
aaoss the foi^ forms of the test. All four forms of the t^t, each form containing the same 60 
core Items and 40 variable items, are administered in each classroom, one form per student. 
Under this system, a fourth of the students in a dassroom wiU take Form 1 of the test, a fourth 
will take Form 2, and so on (see Table 1). * • 



Table 1 




Organization of the Nortli Carolina Test of Chemistry 




60 Core Items 




40 Variable Items 40 Variable Items 40 Variable Items 


40 Variable Items 


100 Items 100 Items 100 Items 
I'orm 1 Form 2 Form 3 


100 Items 
Form 4 



The normative student scores are based on the 60 core items that all forms have in common. 
Curriculum assessment is achieved by combining the results from all four forms, which 
provides an assessment based on the 60 core items + 4(40 variable items), or 220 items in all 



Validity 

The development of a Chemistry achievoment test has two purposes. The first is to obtain 
scores from which inferences may be drawii <»ncKrpig;a\e degrjee o^ success a particular 
student, classroom, school, or school district has ha<d(in mastering the Chemistry curriculum. 
The second is to assess the degree to which the currictdum^has been mastered by students iri^ 
the aggregate. To the extent this; canbe done meaiiiiigfiilly,,test scopes, jnay be said to be valid. 
Thus, one inference drawn from a test score may ibelvaiid, while another rhay riot, 

Theoretidans state that only inferences conceniing test scores may be said to have validity. 
Generally, readers understand this, and, this manual \yiU eiiipidy the convenient shorthand pf 
sneaking about^'test validity" rather than "inferences about achievement drawn from scores 
obtained from tests." 

Test validity is a predominant theme in test development, from the time the idea for a test is 
conceived until the final test scores have been analyzed and mterpreted. For con veriience, the 
various coniponents of test validity will be described as if they were tmique/iiidependent 
components rather than mterrelated parts. Thelirstcomponentof test validity to be desaibed 
will be curricular validity. 

Curricular validity 

If a test is to be used to measure the degree to which a course of sh«Jy has been mastered, the 
first step is to defme the curriculum. Ir the case of Chemistry, that was done through a 
cooperative effort, led by NCDPI Program Services/Division, of Curriculum and Instruction 

Services,involvingcuniculumspedalists, teachers, adnunistrators, university professors, and 
others. The result was a list of nine goals encompassmg 37 objectives (see the Appendix).. 
Supported by expert opinion and a statewide consensus, these goals and objectives were 
approved by the State Board of Education in 1985 as the basis for instruction m Chemistry. 
Curricular validity, the first step in establishing construct validity, was established by this 
mefliod. 

Instructional validity 

Abasiccourseof study may notindude all of the objectives taughtunder various circumstances 
in Chemistry. For example, some advanced dasses may cover material that would be beyond 
the reach of 95% of all Chemistry students. For this reason and several others, it becomes 
important to know just what is being taught in the majority of Chemistry dasses in-the state. 
To determine this, all Chemistry teachers in North Carolina were surveyed in September 1987 
(N = 498). The analysis of results was based on 289 responses, or 58% of all possible responses. 

The Chemistry teachers examined the 37 objectives and noted whether they taught each 
objective every year and whether each objective was basic to instruction. 30 of the objectives 
were rated as basic to instruction by at least 55% of the responding teachers. After deliberation 
by curriculum spedalists, it was determined that the.9 goals and 30 objectives formed tiie basic 
curriculum for Chemistry and the remaining seven objectives would only be used for 
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curriculum assessment, not for students' scores. Tiie objectives are given' in the Appendix 
together with the proportion of teachers judging each one as basic. 

Instructional validity, the second step in defining construct validity> was established by these 
procedures. It defines the inferences that can be drawn from the Chemistry test scores. 

In summary, it was concluded that curricular ?iid instructional validity depended jointly on 
the nine goals and 37 objectives under which they were collected, and that the Chemistry test 
should be built on that foundation. 

Content validity 

Content validity— the degree to which test items reflect the basic instructional program— is a 
quality commonly referenced m evaluating achievement tests. Content validity is built into a 
test from the beginning, and the procedures relating to the content validation of tiie North 
Carolina Test of Chemistry are described below. 

Content validily of the item pool (Phase One). The content validiiy of the item pool was 
defined through a number of operations: 

Ffrst, the item pool for the Chemistry test was created in 1987. It was specified that the pool 
would have 1,221 items, with 33 items per objective. The items were developed by 14 North 
Carolina Chemistry teachers trained in the tschnical aspect? of item-writing. The use of 
classroom teachers from across the state helped to insure that instructional v.»Jidity was 
maintained, since the items would be drawn from their classroom experiences. All total, 1,185 
items were achially written for the Chemistry item pool. 

Second, the item pool was edited for grammar, syntax, psydiometric form, linguisticbias, and 
subject area content. Nine items were deleted from the Chemistry item pool at this time. 

Third, the item pool was analyzed by curriculum specialists and cla.ssroom teachers to assure 
that the items were valid representations of the objectives for whiclt they were written. Each 
item was reviewed by eight classroom teachers— one from each of the eight -educational 
regions across the state. The criteria for evaluating each item included ±e following: 

• conceptual— objective match, fair representation, lack of cultural bias, dear statement, 
single problem, one best answer, common context in foils, each foii credible 

• language— appropriate for age; correct punchiation, spelling, and grammar; lack of 
excess words; no stem/foil dues; no negatives in foils 

• format— logical order of foils; familiar presentation style, print size, and type; correct 
mechanics and appearance; equal length foils 

• diagram— necessary, dean, relevant, unbiased 

164 items were deleted from t'he Chemishy item pool at this time. 



Fourth, the items were collected into 12 test forms for field testing. Although the forms were 
not the final forms of the North Carolina Tpst of Chemistry, they were organized in such a way 
that the objectives were represented equitably across all 6orms. Each form contained 92 to 96 

items, lOof which were common acrossallformsforthepurposeof ability equating should that 
become necessary. 

Fifth, test administration instructions "were written, distribution procedures were organized, 
and administrators were trained to conduct the.test administration. The test administration 
organization used to administer statewide tfsts in NorthCarolina was employed to do the field 
testing. The administration of the field test forms followed the routine eventually expected to 
be used when the test of record was given. 

Sixth, a sample of 8,654 students was selected to take the 12 field test forms containing a total 
of 1,012 items. To insure broad representation, schools were selected from each of the eight 
North Carolina educational regions and were representative of the state based on criteria that 
were judged to be related at least partially to Chemistry ability levels— school performance on 
the 1987 NCT-Algebra H and the 1987NCT-Biology. The 12 field tests were interleaved in all 
students samples, and this produced an even spread of abilit)- across all of the tests. 
Consequently, each item was answered by approximately 721 students (the mmiber of 
students per field test form ranged from 666 to 761). 

Seventh, the field test data were analyized using both the classical psychometric model and the 
one-parameter Rasch model, (results were generated from the BICAL computer program). 
Eighteen statistics were assembled for each item, i.e., p-value, Rasch difficulty index, adjusted 
Rasch difficulty index, standard error of the mean, fit mean-square, item validity (point- 
biserial correlation), and the item characteristic curve groupings. Item bias due to gender or 
ethnicity was examined by computing the partial correlation between the item score and 
gender/ethnicity while controlling for total score. 

The item statistics were submitted to computer analysis using a program designed to scan the 
statistics for an item and print out an appropriate psychometiic notation based on the criteria 
that had been built into the program specifically for Chemistry. An item was classified as "too 
hard" if the p-value was less than .27 or as "too ezsv" if the p-value v^as greater tiian .93. An 
item was said to have "weak prediction" if tiie point-biserial correlation was less than .13. An 
item was said to have an "enb:apment choice," a "mar^al top group," or an "inverted ICC" 
if the item characteristic curve groupings displayed certain irregularities. An item was said to 
exhibit '^gender" bias if the partial correlation witll gender was moreexb-eme than ± .145, and 
to exhibit "ethnic" bias if the partial correlation with ethnicity was more exh-eme than ± .1375. 

The content of Chemistry cannot be represented by a single factor. Therefore, maximization 
of item-total (pomt-biserial) correlations was not a goal of item development. Once an item was 
shown to have at least modest correlation with a corrected total score (.13 or greater) and was 
judged to measure an objective, it was included in the item pool. While this may have reduced 
the potential internal reliability as measured by coefficient alpha, it increased the validity of the 
test by allowing for an objective factor shucture that was not expected to be imitary. 
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This infonnation was placed on an item record, which became the basic document to which all 
other records are referred. Theitem record contains the goal, objective, lUjtorical information, 

acopy oftheitemitself,theitem field-test statistics,and the psychometricnotations.Eachitem 
has a separate item record. 
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items and reached a decision about their curriculaf adequacy. The psychometric and 
curriciilar decisions concerning the item's adequacy for usein test development were included 
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on the item record. 

Of the 1,012 items field-tested in 1988,324(32%) were deleted from theitem pool. This left 688 
items (68%) in the Chemistry item pool for future test development. 

Content validity of the item pool (Phase Two). Due to the large number of items that were 
deleted from the itempool in 1988, itbecame necessary to develop adviitional items for the item 
pool. The greatest need was for items in the higher range of p's. 

First, the mm\ber of items needed per objective was determined from the existing item pool. 
The number needed per objective ranged from 19 to 27 for a total of 770 items. 769 new items 
we'-e written by 11 previously tiained North Carolina Chemistry teachers. 

Second, the additional item pool was edited for grammar, syntax, psychometric form, 
linguistic bias, and subject area content. 51 items were deleted from the additional item pool 
at this time. 

Third, the additional iiam pool was analyzed by curriculum specialists and classroom teachers 

to insure thattheitems were vaUdrepresentaaonsoftheobjectivesfbrwhichtheywerewrit'.2n 
using the procedures previously desaibed. 68 items were deleted from the additional item 
pool at this time. 

Foiirth; the items were collected into 16 sets with 40 items each set (the exception was set #16 
which had 24 new items and 16 repeated items from set #1) for a total of 624 items. These sets 
were used as variable items during the 1990 statewide adminisfration, and the field-test forms 
were mterleaved ^-ith the statewide forms to insure an even spread of abiUty levels. The data 
from these field tests will be analyzed in 1990 and used in constructing tiie 1991 NCT- 

Chemistrjrandftituretests. Of tiienewitems in theitempool,26wereheIdforlaterfield testing 
because they were similar to items contained in the core being used for statewide assessment 
(Core 2). 
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Content validity of the test. After a consideration of the logistics involved, it was decided 
to prepare one complete test (60 coreitems ar.d four sets of 40 variableitems) for administration 
in May 1989, and to develop four additional core tests of 60 items each for use in succeeding 
years. The initial core and the future core tests were based on a random selection of the 30 
objectives rated as basic to instruction in Chemistry, for a total of 60 items (2 per objective) 
randomly chosen from the approved item pool. The four variable sets consisted ofibur items 
per objective for the 30 basic objectives and sbc items per objective for the seven remaining 
objectives randomly chosen from the approved item pool. Thus, the content of the test c :)res 
and the test forms directiy reflected all of the decisions that had been made earlier. 

This method of item selection is a modified domain sampling model, with the various forms 
and cores randomly equivalent. Hie domain sampling model in its pure form is highly 
inefficient because it allows the entiy of items that are grossly inappropriate for normative 
measurement— items that no one can answer or that everyone can answer, or items that have 
psychometric defidendes of a more complex form.-In the modificationnsed here^the domain 
of items was limited to those items that had satisfactory psychometric and curricular 
characteristics. This wasdeterminedbytheanalysesofflieitemfield-testdata, which was used 
to verify the psychometiic and curricular adequacy of the item pool and to direct where item 
revisions should be made. 

After flie test was assembled into forms (60-item core plus a 40-item variable set), the four forms 
were reviewed by one curriculum supervisor and two teachers in each of the eight educational 
regions. The criteria for evaluating each form of the test induded the following: 

• that the content of the test should reflect the goals and objectives taught 

• that the items should be dearly and condsely written, and the vocabulary appropriate 
to the target age levd 

• that the content should be balanced in relation to ethnidty, sex, sodoeconomic status, 
and geographic district of the state 

• that each item should have one and only one answer that is right; however, the 
distractors should appear plausible for someone who has not achieved mastery of the 
represented objective 

The ratings for tiie 1989 NorUx Carolina Test of Chemistiy were average to superior on all of 
the criteria. 

Although tile initial equating of the core tests depended upon the random selection of items 
from tiie item pool, the final equating was based on tiie statistics obtained at the time tii- > first 
test of record was administered (see Table 2). This second psychometric analysis, described 
next, was used to eliminate random differences among the cores and thus facilitate the 
predsion of measurement from onr year to another. 
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Table 2 

Core Development of the North Ccroliha Test of Chemistty 


Core 


Process 


Mean of 
P-values for 
All Items 


SDof 
i—vaiues Or 
All Items 


Sum of 
r-VulUeS or 

All Items 


1 


Design* 


0.592 


0.151 


35.50 




Administer^ 


0.625 


0.158 


37.47 


2 


Design* 


0.589 


0.160 


35.34 




Field Tested^ 


0.584 


0.170 


35.06 




Equate*" 


0.623 


0.139 


37.38 


3 


Des5 rn* 


0.592 


0.160 


35.50 




Field Tested*" 


0.565 


0.174 


33.88 


4 


Design* 


0.591 


0.157 


35.45 




Field Tested*" 


0.580 


0.171 


34.82 


5 


Design* 


0.591 


0.144 


35.43 




Field Tested*" 


0.584 


0.152 


35.04 


"Based on the 1988 Held Test item p-values 
•"Administered in May 1989 
*^quated to Core 1 



Standardization sample. The first North Carolina Test of Chemistry consisted of four forms 
(Core 1/A-D), each form containing the same 60 core items and a unique set of 40 variable 
items. This test was administered to 33^52 North Carolina Chemistry students in May 1989. 
The state norm population comprises these 33^52 students. 

The four additional cores (Cores 2-5) that were developed were interleaved in all student 
samples. A sample of 3,194 students was selected to take the four field core tests. To insure 
broad representation, schools were selected from each of the eight North Carolina educational 
regions and were representative of the state based on a aiterion that was judged to be at least 
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partiaUyrelatedtoChenustryabmtylevds— schoolperfbnnanceonthel988NCT-Bw^^ 
total of 2,370 studentsactuallypartidpafed in theoorefieldtesting(each core was admu^ 
to an average of 593 students). This produced an even spread of ability aaoss all four samples 
(as determined by tiie results of tl\e statewide test administration— Core 1 /Forms A-D. the 
results are as follows: Sample 2, mean = 3t6.86,s =8.32; Sample 3, mean = 36.87,s =8.43; Sample 
4, mean = 36.73/ s - 8.04; Sample 5, mean =36.80, $ = 8.29). The four field core means were not 
equivalent and this is due to random differences in item difficulty that generally occurs 
between field testing and statewide testing. These random differences be eluninated by 

redevelopingthetestsslightly^andthisprocedurevwllbedesaibed in theReliability and Other 
Statistics section. 

1 he lack of agreement between the mean core test scores (Cores 2-5) and the state norm mean 
(Core 1) indicated that the field test samples were not representative of the North Carolina 
Chemistry student population. Uponfurtherexaminationitwasdetermined that this unusual 
finding occurred because not sH schools selected to participate in the field testing had actually 
participated. The resulting score difference based on work with other North Carolina Tests 
and the differences in the Core 1 means for the matched samples compared vdth the Core 1 
statewide mean was approximately 1.5 items (see Table 3 and the discussion above). After 
adjusting for the relative ability of the students in the field core samples, the agreement of the 
mean core test scores supported the view that the initial equating process was successful. 
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Concurrent validity of the test 

When the 1989 North Carolina Test of Chemistry was administered, Chemistry teachers were 
asked to indicate the expected final letter grade for each student in their classes. Figure 1 
displays a comparison of letter grades in Chemistry and the mean NCT-Chemistry core score 
concsponding to each letter grade for the overall student population. The figure corresponds 
dosdy to expectation arid adds to tiie evidence concerning the validity of titie test. 

601 




15-^ « • » ' ^ 

A B C D F 

Anticipated Chemistry Course Grade 

Kguru 1. Comparison of letter grades teachers expected students to 
receive and scores subsequently earned on the 60-item 1989 
North Carolina Test of Chemistry (N = 32,924). 
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Method for Deriving Test Scores 



Item information was available to support the dassicai scoring model and the Rasch scoring 
model. The dassical scoring model gives a unitary weight to each item; a correct choice adds 
one to the total score; an incorrect choice Edds zero. The one-parameter Rasch model also uses 
unitary weighting. (The two- and three-parameter item response models give more aedit for 
answering some itenis correctly and less credit for answering other items correctly. These 
models assimie that each item has a fundamental, "mchanging difficulty level.) 

The classical scoring model was utilized to score the North Carolina Test of Chemistry because 
it is fundamentally sound, simple to use, and easy to interpret. Each student's totc:l core score 
consists of the sum of right answers to the 60 core items. 
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Reliability and Other Statistics 

The descriptiye statistics, the standard errors of measurement, the alternate form reliability 
estimates (correlation between Core 1 and the other four cores field-tested in May 1989), and 
the alpha reliability coefficients for the first statewide administration of the North Carolina 
Test of Chemistry in May 1989 (Core 1 /Forms A-D) and the cores field-tested in May 1989 
(Cores 2-5) are given in Table 3. 



Table 3 

Descriptive Statistics of ttie Nortti Carolina Test of Chiemistry 



Core 



N 



Mean 



SD Median se„ 



Reliability 
Alternate Coefficient 
Form Alpha 



1/A-D 


33,352 


37.47 


8.37 


38 


3.45 


.75, .77, .72, 


2 


589 


35.07 


8.41 


35 








580* 


35.10 


8.37 




3.45 


.75 


3 


597 


33.82 


8.74 


34 








578=" 


33.86 


8.66 




3.35 


.77 


4 


600 


34.70 


7.95 


35 








586=' 


34.90 


7.81 




3.40 


.72 


5 


584 


35.03 


8.42 


36 








569* 


35.13 


8.39 




3.46 


.78 



.83 



.85 



.81 



.83 



"Matched samples of students that took statewide core 1 and one of the field cores (2-5). 
^Average of reliability coefficients obtained from the four matched samples of students (Core 1 

with Cores 2-5). Core 2, Coefficient Alpha = .82; Core 3, Coefficient Alpha = .84; Core 4, 

Coefficient Alpha = .82; and Core 5, Coefficient Alpha = .83. 
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The alternate form reliability estimates have a mean value of .76; and the alpha reliability 
estimates have a mean of .83. While these reliability estimates are lower than those found on 
other tests (NCT-Algebra I, NCT-Biology, and NCT-US Ifistory), they are acceptable. The 
lower estimates can be attributed to the restricted range of ability of the students taking the 
North Carolina Test of Chemistry. Chemistry is not a required course for high school 
graduation and, therefore, students taking Chemistry are a select group of students when 
compared to the general populationof high school students taking other Nortii Carolina Tests. 
The 1989 core scores are symmetrically distributed about a mean of 37.47, or 62% correct (see 
Figure 2). 



1.7 




Score on 60-itein NCT-Chemistry Core 



Figure 2. Frequency distribution of scores on the 60-item 1989 North 
Carolina Test of Chemistry— Core 1/Forms A-D (N = 33,352. 
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For practical piyposes, the proper measure of reliability is the alternate form reUability. The 
calculation of this statistic requires that two or more equivalent forms be developed. The "older 
alternate form reliability procedure required the devdppment of one form, whid^ Was then 
Moned'' to obtain a second, alternate form of the test. A judicious selection of alternate items 
was recommended to prevent direct memory transfer from the administration of one t^t to its 
alternate form. But the possibility remained that ierrors of selection in the first form would be 
duplicated m the second form. A newer procedure requires that the tests be truly equivalent— 
that is, that two or more tests be developed in exactiy tiie same way, but independent ot* one 
another, m permits Hie reliability coefficient to reflect any random errors in section made 
in the development of either of tiie test forms. 

The alternate forms developed for the Nortii CaroUna Test of Chemisb-y reflect this newer 
procedure. That is, each test form is developed from tiie domain of items in exactly tiie same 
manner. Any failure of the alternate form reUability to be 1.00 reflects: 

• brait instability not followmg from mati^ratipn or insh-uction 

• insbrument instability resulting from faSlible test development procedures 

• administrative instability reflecting different testing occasions 



r 



The square of the alternate form reUabiiify coefficient accurately reflects tiie maximum 
proportion of variance one can legitimately expect to predict from tiie adminisb-ation of the 
Nortii CaroUna Test of Chemisby (r2 = .'832 = .69) when test scores are compared aaoss time 
or witii otiier measures of shident abiUties or personaUty braits that have similar reUabilities. 
In brief, ihe alternate form reliabiuty coefficient is tiie statistic to use when correcting for 
attenuation. 

Of special significance to Hie comparison of stiidents scores across time is tiie equivalence of 
ihe four future core tests (Cores 2-5) to ihe first core test of record (Core 1 administered May 
1989). An equipercentile analysis was made of ihe relationship.of ihe four ftihire core tests to 
pre 1 (core test of record). Matched samples of shident scores were obtained for each of the 
four fuhire cores tiiat consisted of shidenf s scores on ihe statewide core and one of ihe four 

fuhirecores(580shidentswereadministeredCorelandCore2;578studentswereadministered 
Core 1 and Core 3; 586 shidents were administered Core 1 and Core 4; and 569 shidents were 

admmistered Corel andCore5-seeTables2and3). To makefile equipercentile comparison, 
ihe mean of a block of scores on Core 1 within groups of five successive percentile points was 
taken to compare vnih ihe mean of a block of scxDres on Core 2 within ihe same five percentile 
pomts. This procedure yielded twenty reasonably reUable points of comparison. This 
procedurewasrepeatedforthematchedsamplesfi:omCores3-5. Each of tiiesemidedle scores 
was adjusted for differences in relative abiUty of the four field core samples by adding 1 5 
pomts. The results of the equipercentile comparisons of Core 1 vnih Cores 2-5 are presented 
m Figures 3, 5, 6, and 7. 
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In Figures 3, 5/ 6, and 7, the differences of the data points from perfect agreement (a slope of 
1 .00) are small. These differences could be adjusted statistically by providing a separate set of 
norms for each test. A simple and efficient alternative is to redevelop the core tests slightly so 
that even thesmall differences disappear. Wifii tihis technique, a single norms table can be used 
for all five core tests (20 test forms). To accomplish this transformation, the test developer had 
statistical infomvatipn available on the 60 items in Core 1 plus the 160 variable items— a total 
of 220 items for whidi comparable psychometric data were available across all four future core 
tests. 



60 
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Raw Scores in Semi-decile Classes 
Perfect Agreemsnt + Field Core 2 



Figure 3. Equipercentile comparison of the 1989 Norfli Carolina Test of 
Chemistry—Core 1 /Forms A-D and Field Core 2 (adjusted for 
tmrepresentativeness of flie sample). 
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The results of the equating for Core 2, employed in 1990, are presented in Table 2 and Figure 
4. The required.changes were minimal (the substitution of 4 items). 




Raw Scores in Semi-decUe v'iosses 
Perfect Agreonent + Field Cbre 2 - Adj 

Figure 4. Equipercentile comparison of the 1989 North Carolina Test of 
Chemistry— Core 1/Forms A-D and Field Core 2 Equated 
(adjusted for unrepresentativenes? of the sample). 



Similaradjustments based on the 1989 administration of Core 1 and thefield testadministration 
of Cores 3-5 will be made to the third, fourth, and fifth core tests as needed in the future. The 
adjustments to the core tests will assure continuity of the norms table for future years while 
providing new test items each year. The new test ih^ms prevent the loss of confidentiaUty, and 
therefore validity, that occurs with the continued xifje of the same items. Student scores have 
a common reference point from 1989 onward, barring changes hi the definition of the basic 
instructional program. 
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Raw Scores in Semi-decile Classes 
Perfect Agreement + FiddCwe3 



Figure 5. EquipercentUe comparison of the 1989 North Carolina Test of 
aiemistry-<:ore 1/Fonns A-D Mid Field Core 3 (adjusted for 
unrepresentativeness of the sarnple). 
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Raw Scores in Semi-decile Classes 
Perfect Agreement + Field Core 4 



Figure 6. EquipercentUe comparison of the 1989 North Carolina Test of 
Chemistiy--Core 1/Forms A-D and Field Core 4 (adjusted for 
unrepresentativeness of the sample). 



ERIC 



Technical Manual 



Page 



17 



60 




lb ' ^ ^ ^ 

Raw Scores in Scmi-dccilc Classes 
Perfect Agreement + Field Core 5 



Figure 7. Equipercentile comparison of /ihe 1989 North Carolina Test of 
Chemistry— Core 1 /Forms A-D and Field Core 5 (adjusted for 
uiurepresentativeness of the sample). 
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Curricular Assessment 

Ilius for in this manuaa the 60-item core test has been discussed as if it made up the entire test. 
In actuality/the four test-forms of the 1989 North CaroUna Test of Chemistrjrdonsist^ of a60- 
item core and 40 additional items'thaf varied aaoss the four forms (see Tabiel): llie.variable 
itemi-were not intovded fo obhtribute to individual f/mdent scores, b]dt ra^^l]^ fo (^culum 
assessment. Each variabieitein was answered^b^ * 

At the dassrobm levels 220 items are answered during each test admjjus&atioiil)y ah average 
of five studeiits: Thi§ procedure provides a database of sbc items p^ bbj^veici^)^ five 
students. Froin this database of information, evidence of liow yarioUiS portions 6f%e 
curriculum are beingmastered in the dassroonimay be drawn. At the sdiobl; ^ool distfict, 
and state level, the 220 items are answered by armudvlarger number of smdentsrover 8,000 
students per item. This assures a more stable measurement, but does not ihciude a. larger 
number of objectives or items. The accumulation of item and objective informatiohi^eprads 
upon measurement across isuccessive years. ' 

Themeasurementaffoidedby tiie 160 variableitems is critical to assessingcurriculum mastery 
at the dassroom, school, school district, and state levels. Each year of test admihistratibn adds, 
to the database and gives a more detailed and comprehensive picture of dnriailum succe^. 
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Content of The Test 



The North Carolina Test of Chemistry Is objective-referenced; that is, its reference is to a 
domain of objectives. This domain is mapped over a dpmaiii of items, where the items reflect 
the objectives, equal in kind and number except for random fluctuations. Hie Chenistry tests 
were designed to achieve an even assessment across all objectives; in short, each objective was 
to be represented by the same number of items. lius design is consistent with the concept of 
a domain of objectives mapped over by a domain of items, AlUiough the objectives have unit 
weighting, the goals are weighted by the number erf objectives assigned them. R-om 
empirical analyses, this is a traditional aspect of curricultmi development: the more important 
agoalis considered tobe,thegreaternumber of objectives thatwillbedevelopedforit. Thus, 
an tmderlying system of weights exists for the curricular goals. 

The Appendix lists each goal and objective aid the numerical item repr^entation for each 
objective as it appears on tiie 1989 North Carolina Test of Chemistry (Forms A-D). In addition, 
the proportion of teachers rating each objective as basic to instruction in the Chemistry 
curriculum is listed. 

Tables 4-1 through 8 list the difficulty level for all items on the 1989 North Carolina Test Of 
Chemistry (Core 1 /Forms A-D) and for all items on the four future core tests (Cores 2-5) in 
terms of pvalues (proportion of all students answering the item correctiy). 
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Table 4-1 

Item DifficuHy by Item Number for the NCT-Chemistry-Core 1 



ltem# P-value ltem# P-yalue Item# P-value 



1 


.59 


21 


.79 


41 


.55 


2 


.62 


22 


.71 


42 


.59 


3 


.68 


23 


.72 


43 


.85 


4 


.70 


24 


.74 


44 


.48 


5 


.52 


25 


.61 


45 


.81 ■ 


6 


.71 


26 


.61 


46 


.55 


7 


-77 


27 


.65 


47 


.54 


8 


.68 


28 


.63 


48 


.39 


9 


.86 


29 


.57 


49 


.38 


10 


.61 


30 


.75 


50 


.55 


11 


.86 


31 


.57 


51 


.66 


12 


.68 


32 


.36 


52 


.32 


13 


.57 


33 


.34 


53 


.51 


14 


.61 


34 


.39 


54 


.32 


15 


.56 


35 


.47 


55 


.59 


16 


.62 


36 


.42 


56 


■ .35 


17 


.54 


37 


.46 


57 


.79 


18 


.50 


38 


.32 


58 


.46 


19 


.75 


39 


.31 


59 


.85 


20 


.67 


40 


.67 


60 


.77 
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Table 4-2 

Item Difficulty by Item Number for the 1989 NCT-Chemistry— Variables A-D 



ltem# 

Al 

A2 

A3 

A4 

A5 

A6 

A7 

A8 

A9 

AlO 

All 

A12 

A13 

A14 

A15 

A16 

A17 

A18 

A19 

A20 



P-value 

.90 
.46 
;69 
.71 
.43 
.79 

32 
.63 
.69 
.59 
.51 
.71 
.74 
.73 
.47 
.67 
37 
.67 
.69 



ltem# 

Bl 

B2 

B3 

B4 

B5 

B6 

B7 

B8 

B9 

BIO 

Bll 

B12 

B13 

B14 

B15 

B16 

B17 

B18 

B19 

B20 



P-value 

.92 

.70 

.39 

.47 

.94 

.59 

.80 

.73 

.82 

.43 

.55 

.86 

.70 

.36 

.48 

.80 

.58 

.32 

.3? 

.52 



-ltem# 

CI 

C2 

C3 

C4 

C5 

C6 

C7 

C8 

C9 

CIO 

Cll 

C12 

C13 

C14 

C15 

C16 

C17 

C18 

C19 

C20 



P-value 

.95 
.33 
.46 
.80 
.56 
.60 
.65 
.43 
.83 
.80 
.91 
.74 
.71 
.57 
.64 
.48 
.57 
.35 
.50 
.30 



ltem# 

DI 

D2 

D3 

D4 

D5 

D6 

D7 

D8 

D9 

DIO 

DU 

D12 

D13 

D14 

D15 

D16 

D17 

D18 

D19 

D20 



P-vdIue 

.95 
3S 
.73 
.95 
.77 

;74 

.64 
.49 
i58 
.45 
.48 
.39 
.66 
.44 
.63 
.74 
.37 
.67 
.72 
.30 



23 



30 



Table 4-2 (continued) 














Item Difficulty by Item Number for the 1989 NCt-Chemistry- 


-Variables A-D 








ltem# 


P-value 


item # 


P";value 


iTi^rn IF 


ri 1 ' - 

r*yalue 


irem F 


r-vaiue . 


A21 


.49 


B2i 


.35 




•59 


U21 


.43 


A22 


.39 


B22 


.40 




•45 




.51 


A23 


.48 


B23 


.42 




AO 

.48 


L)23 


.75- 


A24 


.42 


B24 


.42 




Af\ 

.49 


L/24 


■57. 


A25 


.47 


B25 


.38 




.33 


D25 


,45 


A26 


.49 


B26 


34 




.43 


D26 


.31 


A27 


.41 


B27 


41 




.43 


P27 


.41 


A28 


.40 


B28 


.65 


vJZo 


.31 


D28 


.46 


A29 


.56 


B29 


42 




.44 


D29 


;32 


A30 


.61 


B30 


.84 




.51 


D3J 


.74 


A31 


.4b 


B31 


.64 




>f o 

.42 


D31 


•47 


A32 


.41 


B32 


.91 




.54 


D32 


.52; 


A33 


.55 


B33 


.62 




.79 


■' D33 


.51 


A34 


.49 


B34 


.43 


C34 


.66 


b34 


.42 


A35 


.40 


B35 


.47 


C35 


.38 


D35 


.38 


A36 


.40 


B36 


.34 


C36 


.40 


D36 


.38 


A37 


.42 


B37 


.39 


C37 


.45 


D37 


.44 


A38 


.47 


B38 


.37 


C38 


.64 


D38 


.46 


A39 


.46 


B39 


.40 


C39 


.61 


D39 


.87 


A40 


.65 


B40 


.62 


C40 


.63 


D40 


.71 



31 



32 
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Item Difficulty by Item Number for ttie NCT-Chemlstryr- 


-Core 2 




II wl 1 1 TT 








ltem# 


P-value 


1 
X 




71 

•Cx 




41 


.35 


9 




JUL, 




42 


.72 


3 




23 




43 


.40 


4 


53 


24 




44 


.36 


5 


90 


25 


82 


45 


.66 


6 


62 


26 


46 


46 


.7Q 


7 


52 


27 


60 


47 


.39 


8 


64 


28 




48 


;50 


9 


71 


29 


77 


49 


.32 


10 


ftO 


30 




50 


.55 


11 


70 
•/u 


31 




51 


.42 


19 


70 
•/J 


39 




52 


.49 


13 


.84 


33 


«36 


53 


.41 


14 


.4U 




.DO 


54 


.34 


15 


.74 


35 


.56 


55 


.59 


16 


.67 


36 


.44 


56 


:53 


17 


.56 


37 


.60 


57 


.82 


18 


.46 


38 


.41 


58 


.51 


19 


.36 


39 


.79 


59 


.85 


20 


.77 


40 


.67 


60 


.58 
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Tabled 

Item Difficulty by item Number for the NCT-Chemlstry-Core 3 



ltem# 


P-value 


ltem# 


P-value 


ltem# 




P-value 

- 


1 


.45 


21 


.52 


41 


.64 


2 


.66 


22 


.84 


42 


.40 


3 


.44 


23 


.84 


43 




4 


.81 


24 


.67 


44 


.63 


5 


.53 


25 


.61 


45 


.52 


6 


.56 


26 


.43 


46 


.78 


7 


.57 


27 


.83 


47 


.72 


8 


.57 


28 


.59 


48 


.50 


9 


.44 


29 


.72 


49 


.38 


10 


.64 


30 


.52 


50 


.55 


11 


.79 


31 


.41 


51 


-.32 


12 


.72 


32 


.84 


52 


.38 


13 


.84 


33 


.59 


53 


.51 


14 


.53 


34 


.31 


54 


.33 


15 


.66 


35 


.42 


55 


.43 


16 


.53 


36 


.45 


56 


.74 


17 


.77 


37 


.42 


57 


.88 


18 


.53 


38 


.56 


58 


.88 


19 


.68 


39 


.33 


59 


.73 


20 


.66 


40 


.57 


60 


.86 
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Table? 

Item Difficulty by Item Number for the NCT^hemlstry— Gore 4 



ltem# 


P-volue 


ltem# 


P-value 


ltem# 


P-valu© 


1 


.45 


21 


.70 


41. 


M 


2 


.61 


22 


.in 


42 


59 


3 


.41 


23 


.75 


43 


.62 


4 


.69 


24 


.87 


44 


.66. 


5 


.45 


25 


.4 


45 


. .57 


6 


.78 


26 


.74 


46 


.59 . 


7 


.51 


27 


.77 


47 


.36 


8 


.81 


28 


;56 


48. 


.54 


9 


.69 


29 


M 


49 


,56 


10 


.90 


30: 


.40 


50 


.55 


11 


.86 


31 


.59 


51 


.45- 


12 


.70 


32 


.40 


52 


.35 


13 


.73 


33 


.62 


53 


.48 


14 


.55 


34 


.38 


54 , 


.32 


15 


.60 


35 


.41. 


55 


,36 


16 


.72 


36 


.56 


56 




17 


.87 


37 


.65 


si 


;81 


18 


.53 


38 


.30 


58 


.44 


19 


.67 


39 


;84 


59 


.74 


20 


.54 


40 


.67 


60 


.73 
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Item Difficuliy by Item Number for the NCT-Chemistry- 


-Core 5 




ITem rF 


r~vaiu© 


lT©m iF 


D_v//«»t| Id 


ltem# 


P-volue 


1 


A A 


01 
Zl 




41 


.36 


2 


CI 

.bo 


2Z 


/A 
.W 


42 


.45 


o 


./o 


ZD 


.0/ 


43 


.55 


A 

4 


•71 
./I 




70 
.iO 


44 


.35 




CI 

.Dl 


25 


.oZ 


45 


.39 


0 


CO 
.DO 


Zo 


•74 


46 


.91 


7 


.63 


27 


.0/ 


47 


.75 


o 
o 


./u 


Zo 


QC 


48 


.59 


V 


CI 


zy 


cc 


49 


.47 




QQ 
•OO 


on 


>11 
♦41 


50 


.55 




Jo 


oi 


.4/ 


51 


.39 


10 

IZ 


CA 

.xy± 


ao • 
oZ 




52 


.68 


13 


.54 


33 


.40 


53 


.48 


14 


.62 


34 




54 


.63 


15 


.67 


35 


.64 


55 


.50- 


16 


.80 


36 


.49 


56 


.37 


17 


.48 


37 


.41 


57 


.59 


18 


.53 


38 


,41 


58 


.69 


19 


.56 


39 


.58 


59 


.64 


20 


.58 


40 


.76 


60 


.56 
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test Norms 



Students who answer all. 60 of the Chemistry core items coirectly could be assunied to be 
excellent Chemistry students; K everyone answered all of the items , correctly, -however, a 
dtfferentihterpretationwoiddhavetobeplacedonthescores. Atsornepoint,s^^ 
a reference group grounded in the experience of all students, In some respect, at least, 
everythingisgoodnrbadbycompaiisoii. Norms tabl^proyidethatreference. Givenanorms 
table, a student's score can be compared with other students' scores. 

Norms tables commonly have two points of reference: a scale of percentiles and a scale of 
standard scores. The former permits the location of ^ score wittiin percentile ranks; thus a 
student is said to have exceeded the performance of of the students in the norm group (in 
this case. Chemistry students taldng the North Carpiina test of Chemistry in May 1989). The 
latter/Standard scores, permits the location of a score witiiin normdly-distributed standard 
score'i. This reference is appropriate if the student abilities are believed to be normally 
distributed. In a normal distributipn, raw scores are given greater and greater weight as they 
diverge from the mean in eiflier direction. 

The choice of a metric for flie standard score is arbitrary. To avoid inappropriate and confusing 
comparisons with some of the more common metrics, such as thos6 employed in IQ scorco Oir 
NCE scores, a metric having a mean of 50 and a standard deviiation of 10 was chosen. Most 
curriculum research studies involving the summation of scores will find the standard score to 
be the statistic of choice. 

The norms table for student scores on the North Carolina Test of Chemistry is given in Table 
9. These scoies set a baseline of comparison for present and future achievement in Chemistry. 
Thus, a student score in 1990, 1991, and future years can be referenced to the scores of all 1989 
Chemistry students in Norfli Carolma. 

In summary, flie utility of a test is its statistical equivalence of core tests from year to year, its 
broad sampling of the curriculum aaoss time, and its initial norms table. 
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Table 9 

Norms for Student Scores on the North Carolina Test of Chemistry 



Raw Score 



Perc\'>ntiie 



Standard Score'' 



ou 


99 


76.8 




99 


75.6 


00 


99 


74.4 


0/ 


99 


73.2 




99 


72.0 


oo 


99 


70.8 


Oft 


98 


69.6 


OO 


97 


68.5 


oz 


96 


67.3 


01 


95 


66.1 


ou 


Ail 
94 


64.9 


AO 


92 


63.7 


AQ 


OA 

89 


62.5 




87 


61.3 




84 


60.1 




01 


58.9 




// 


fry rr 

57.7 


43 


/o 


56.5 


42 


07 


00.4 


41 


00 


C4 O 

54.2 


40 


ou 


CO A 

53.0 


39 


00 


ct o 

ol.o 


38 


01 


CA ^ 

50.6 


37 




>IA >l 

49.4 


36 


AO 


AO O 

48.2 


35 


Oo 


47.0 


^4 


c54 


45.8 


oo 


OA 

30 


44.6 


o^ 


A^ 

26 


43.5 


Oi 


22 


42.3 


Ou 


f A 

19 


41.1 


90 


16 


39.9 


Zo 


13 


38.7 


27 


11 


37.5 


26 


9 


36.3 


25 


8 


35.1 


24 


6 


33.9 


23 


5 


32.7 


22 


4 


31.5 


21 


3 


30.4 


20 


2 


29.2 


19 


2 


28.0 


Than 19 


1 





"Adjusted to a mean of 50 and a standard deviation 10.0. 
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Test Content— Item Representation by Goal and Objective 



% Teachers 
No. Items Reporting 

Goal/Obj Description 1989 osBosIc^ 

Goal 1 The learner will have an understandiT g of the 
lustory, scope, basic concepts, and techniques 
related to the study of chemistry. 

1.1 Know and apply accepted methods, processes, 6 97.18 
and procedures for conducting scientific study. 

1.2 Know the properties of matter and energy. 7 97.54 

1.3 Know the concept of conservation of matter and 6 99.30 
energy. 

Goal 2 The hamer will understand systems of 

classification of matter, nuclear, physical, and 
chemical changes. 

2.1 Know various desaiptive classifications of 7 97.16 
matter. 

2.2 Know the basic chemical concepts of atoms and 6 99.65 
molecules. 

2.3 Know the concepts of elements, compounds, and 6 99.30 
mixtures. 

2.4 Know various types of nuclear changes. 6 44.60 

2.5 Know about physical changes such as phase 6 97.89 
changes and the characteristics of these changes. 

2.6 Have knowledge of the nature and evidence of 6 98.59 
chemical changes. 

Goal 3 The leamer will have an understanding of 

descriptive chemistry and periodic properties of 
eleiTients. 

3.1 Know descriptive terminology pertaining to 6 95.74 
atomic mod ''*; and configurations of electrons. 

3.2 Know the " *n and nature pf the periodic 7 95.74 
properties or the elements, and the utility of the 

periodic table. 



'Percentage of North Carolina Chemistry teachers rating the objective as basic to instruction in Chemistry. 
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Goal/Obj Description 



% Teachers 
No. Items Reporting 
1989 as Bdsic^ 



Goal 4 The learner will understand concepts and 

techniques of measurenw^nf and computation as 
they relate to chemistry. 

4.1 Know how to measure accurately length, area, 7 97.52 
volume, mass, weight, temperature, and time, 

and record the measurement as precisely as the 
measuring devices permit. 

4.2 Know how to use scientific notation. 6 97.51 

4.3 Have a knowledge of mafliematical operations 6 97.87 
involving manipulation of xmits and unit 

conversions. 

Goal 5 The learner will have an understanding of 
stoichiometry and kinetic molecular theory. 

5.1 Know how to construct and use chemical 7 99.65 
formulas and equations. 

5.2 Know how to use the mole concept. 6 96.47 

5.3 Know how to make calculations involving 6 95.41 
stoichiometry given a periodic table and a 

calculator. 

5.4 Know how to make calculations for the 6 84.04 
prediction of the beha dor of gases. 

Goal 6 The leamer will understand fundamental 

principles related to chemical reactions, kinetics, 
and thermodynamics. 

6.1 Know the concept of oxidation-reduction. 5 63.96 

6.2 Have knowledge of basic principles in 7 29.43 
electrochemistry. 

6.3 Have knowledge of various energy effects in 6 57.65 
chemical reactions. 

6.4 Know factors that affect the rate of a reaction. 5 74.20 

6.5 Know the concept of dynamic equilibrium. 6 55.87 
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Goal? The learner will have an undastanding of tile 
properties of electrol}^ solutions. 

7.1 Know tile importance of adds, bases, and salts in 3 81.21 
industry and in the home. 

7.2 Know filenames and formulas of selected adds, 7 96.47 
bases, aijd salts. 

7.3 Know physical characteristics and chemical 6 90.11 
properties of solutions of adds, bases, and salts. 

7-4 Know selected expressions of conrentration of 3 84.10 
solutions. 

7.5 Have a knowledge of the phenomenon of 5 77.94 
ionization. 

7.6 Have a knowledge -of add-base equilibria and 6 6429 
pH. 

7.7 Have a knowledge of solubility equilibria. 7 37.99 

Goal 8 The learner will have an understanding of tiie 
prindples, reactions, and related compounds 
studied in organic chemistry. 

8.1 Have knowledge of chemical properties, physical 6 67.62 
forms, and atomic structure of carbon. 

8.2 Have knowledge of hybridization and its 5 33,09 
relationship to bonding and molecular geometry. 

8.3 Have knowledge of hydrocarbons. 6 49.82 

8.4 Have knowledge of major hydrocarbon 6 28.42 
substitution products. 

8.5 Know that activities of living things involve 6 51.07 
chemical reactions. 

Goal 9 The learner will have an understanding of the 
relevance of current topics in chemistry. 

9.1 Have knowledge of the relevance of current 6 58.57 
topics in chemistry. 

9.2 Be aware of careers available in chemistry. 7 58.87 
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